-
Notifications
You must be signed in to change notification settings - Fork 150
feat: add Active Agent monitoring plugin #754
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR adds a new plugin for monitoring Active Agent, a framework for building AI agents. The plugin provides both exception tracking and insights collection capabilities similar to other Honeybadger integrations.
- Adds a new Active Agent plugin with error handling and event subscription
- Configures insights collection for Active Agent events (prompts, embeddings, streaming, tool calls, and processing)
- Adds configuration option to enable/disable Active Agent insights
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| lib/honeybadger/plugins/active_agent.rb | New plugin that registers Active Agent monitoring, subscribes to Active Agent notification events, and provides hooks for exception reporting |
| lib/honeybadger/config/defaults.rb | Adds active_agent.insights.enabled configuration option (defaults to true) for controlling automatic data collection |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@TonsOfFun do you have a sample Rails app that uses all of Active Agent's features by any chance? That would be useful for testing (I made a sample app, but it doesn't do tool calls, embeds, or streaming, and I'm not sure how to simulate errors). |
|
https://github.com/TonsOfFun/writebook this should have a few feature examples you can check out. Let me know if you have any questions! |
|
The main application I'm using ActiveAgents for is also supported by HB; I can test some of this stuff in prod once it's ready. |
Amazing, I'll let you know when it's ready to test! @TonsOfFun I got your writebook running, but I'll need to upgrade active agent to 1.0 since that's what I'm targeting for telemetry (I'm assuming that's the right target). |
Currently AA is pretty agnostic on the exception handling. Prior to v1, there was some generic retry logic. This has been removed in favor of native retry logic built inside of the LLM provider's native gems. So right now the errors bubble and report to HB just like any other error. It's not clear to me how much AA needs/should be involved with error handling. A lot of it seems out of scope to me, but I'm also getting errors from the MCPs bubble back to the LLM which effectively blow the entire conversational stack. Admittedly I don't really use HB insights at the moment, some people on my team do; but this area likely the area exploration when it comes to errors, since this can make errors pretty expensive. |
Yeah, that's what I'm thinking too. I'm going to research some of the callbacks and integration points — we may be able to add some additional context to errors that happen in Active Agent actions (similar to how we do for Action Controller in Rails). But it sounds like reporting the errors themselves should be the responsibility of our standard Rails middleware and/or the user if they are handling errors internally. |
f5f2101 to
9f87188
Compare
This pollutes the diff. We can update them later if necessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
stympy
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking of making the same change you did (adding RailsSubscriber) while I was working on the Flipper plugin. 👍
This PR adds Active Support instrumentation to capture the following Active Agent events:
prompt.active_agent,embed.active_agent(excluding.provider.active_agentevents)stream_open.active_agent,stream_close.active_agent,stream_chunk.active_agenttool_call.active_agentprocess.active_agentThe raw events look like this:
Open questions
.provider.active_agentevents for individual API calls in multi-turn conversions? If so, we may want to unsubscribe from prompt.active_agent and embed.active_agent events to avoid duplicates. I'm not sure if that would prevent us from showing the total duration for prompts, though.Do we need special handling for exceptions? Honeybadger reports unhandled exceptions in jobs and web requests automatically, so additional error reporting my be unnecessary—unless we can provide additional context or catch exceptions that are rescued internally. (For example, Sidekiq provides its own API for error handling.)We'll rely on Honeybadger's built-in exception handling.Do we want to collect metrics? We can create charts from the the values in the instrumentation events, but rolling up metrics can sometimes make charting easier (and users can fine-tune the data they wish to send).We may add metrics in the future if users need them (e.g. if the current events are insufficient for monitoring things like duration, usage, etc.).Known issues
prompt.active_agentevent includes the full prompt text, which may result in higher data ingest. It also includes it twice, once in the top-levelmessagesfield, and again in theparameters.messagesfield. We could remove one or both of these in the future, but it would be better if Active Agent resolves the duplicate fields. Do users want to be able to inspect themessagesdata when monitoring prompts?prompt.active_agentevent also does not include theusagewhich Active Agent describes in the docs.stream_chunk.active_agentevents. We're currently including them, but could remove them in the future. Are they useful?TODO
Create a Honeybadger automatic dashboard templateWe'll do this later.Reference